Migrating from Databricks
Teams come to us with a Databricks estate they already trust: hundreds of notebooks, a Hive Metastore or a Unity Catalog full of tables, jobs wired together with dependencies, and a dbutils call on almost every page. They want that estate running on their own infrastructure, inside their own network, without a rewrite.
We have a solution for that move, and this section describes it honestly.
Yeedu runs open-source Apache Spark and open table formats, so the Spark half of a Databricks codebase carries across untouched. The Databricks-proprietary half is the interesting part. We ship a dbutils compatibility layer so dbutils.fs.ls, dbutils.secrets.get, and dbutils.widgets.get keep working, and we ship a migration utility that inventories a Databricks workspace, moves the metastore, rewrites the code that needs rewriting, and emits Airflow DAGs that reproduce your job graph.
Some things don't carry across. We say which, on What Is Supported, by name.
What the move involves
Six phases. Our tooling drives four of them and assists with a fifth.
| Phase | What happens | Who does it |
|---|---|---|
| Inventory the workspace | Read system.information_schema and write out every table, view, function, volume, and dependency edge | Automated |
| Assess what maps cleanly | Scan each job's code for dbutils calls, Unity Catalog references, unsupported SQL, and identity-bound functions; validate that the target principal can actually read what the code touches | Automated |
| Migrate the metastore | Move Hive Metastore managed tables, external tables, views, and grants into Unity Catalog | Automated, with a manual credential-mapping loop |
| Convert and migrate jobs | Download job files, rewrite Databricks-specific code, generate Airflow DAGs, push files into a Yeedu workspace | Automated |
| Handle Databricks-specific APIs | Decide what to do about DBFS mounts, multi-language notebooks, and any dbutils namespace we don't implement | Human judgement |
| Validate | Run the migrated jobs and compare results against the original Databricks runs | Human judgement |
The two phases at the bottom are where the real work sits, and no tool can do them for you. Everything above them is mechanical, so we automated it.
The parts we automate
Our utility processes jobs in parallel, downloads every notebook and Python script a job references, and passes each file through a chain of rewrites. DeltaTable becomes YeeduDeltaTable. hive_metastore becomes your target catalog name. Table references inside OPTIMIZE, VACUUM, and CACHE TABLE get backticked, including inside f-string SQL. Every dbutils call is logged and checked against the set we support.
It also walks the dependency chain for every Unity Catalog object your code touches, using the view-edge graph collected during the inventory phase, then checks USE_CATALOG, USE_SCHEMA, and SELECT for the principal that will run the job on Yeedu. Where a grant is missing, it writes the GRANT statements into a notebook you can run in Databricks.
The output is a per-job verdict. job_readiness_report.csv marks each job READY or not, with the specific reason.
The parts that need a person
Databricks mount points are the usual one. A path like /mnt/datalake/orders means nothing in Yeedu, and files under /dbfs/ are skipped rather than downloaded, because there's no DBFS to download them from. Someone has to decide what those paths become.
Multi-language notebooks are another. A Yeedu notebook speaks one language, so a Databricks notebook that switches between %python and %scala cells has to be split.
And then there's identity. Functions like current_user() and is_member() resolve against whoever ran the query in Databricks. After migration they resolve against whoever runs it in Yeedu, which may be a different principal entirely. Our utility flags every use of them. It cannot tell you whether the new answer is the one your logic depends on.
Where orchestration lands
Yeedu doesn't ship a native multi-task workflow engine the way Databricks Jobs does. Orchestration runs on Apache Airflow or Prefect, and we publish operators for both:
Our utility generates the Airflow DAGs for you, straight from the Databricks job definitions. Multi-task dependencies become task dependencies. for_each_task becomes a mapped task. condition_task becomes a branch. Quartz schedules keep their semantics through a custom timetable. Details are on Job Migration.
Reading the rest of this section
| Page | What it answers |
|---|---|
| Feature Comparison | How Yeedu stacks up against Databricks across ten capability areas, with the caveats spelled out |
| What Is Supported | The exact dbutils surface, the SQL that doesn't carry across, and the Databricks features with no Yeedu equivalent |
| Migration Utility | What the utility does across its three phases |
| Job Migration | How a Databricks job becomes a Yeedu job plus an Airflow DAG |
Every capability claim in this section comes from the Yeedu dbutils implementation, the Databricks migration utility source, or the Yeedu platform comparison. Where the Python and Scala implementations of dbutils differ, we call the difference out.